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Work integrated learning (WIL) educators using reflective practice to facilitate student learning require a set of 
standards that works within the traditional assessment frame of Higher Education, to ascertain the level at which 
reflective practice has been demonstrated. However, there is a paucity of tested assessment instruments that provide 
guidance for measuring student demonstrations of reflective learning in WIL. This paper provides a preliminary 
exploration (pilot) of the reliability and validity of a WIL placement rubric for grading reflective practice across an 
alternate WIL placement-//^ context, for the purpose of testing the potential transferability of the rubric. The study 
supports earlier research that suggests that inter-assessor shared understanding of standards are crucial for achieving 
reliable and valid results. The rubric's value is in providing a foundation set of standards that can support WIL 
educators to expedite discussions for developing and using assessment tools grounded in reflective practice theory for 
grading student achievement across a range of WIL placement -like learning contexts. (Asia-Pacific Journal of Cooperative 
Education, 2016, 17(2), 135-147) 
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WIL placement curricula incorporating assessment of reflective practice are gaining traction 
in higher education because they are seen to enable and improve deep learning (Boud, Keogh 
& Walker, 2013; Correia & Bleicher, 2008; Kember, McKay, Sinclair, & Wong, 2008) and 
enhance employability (Yorke, 2008). Work placements (internships, practicums, industry 
based learning (IBL), and service learning, to name a few) are the most commonly recognized 
learning activities of WIL. Less well recognized work scenarios, but still with a focus on 
praxis (application of authentic course-specific knowledge and skills usually undertaken in 
industry/community) may include consultancy projects, study tours and intensive field trips. 
In this paper, we use the term placement-ELe experiences to purposely broaden traditional 
conceptions of placement in WIL curricula. 

WIL educators have accepted the contribution that personal reflection on WIL experiences 
make to strengthening professional practice (Schon, 1983; Boud, Keogh, & Walker, 1984; 
Henderson, Napan, & Monteiro, 2004; Higgins, 2011; Wingrove & Turner, 2015). This has led 
to increasing emphasis on developing students' reflective skills in WIL placemen t-like 
curricula, however a rise in empirical evidence pertaining to effective assessment tools for 
pinpointing standards of reflective practice achieved in student assessment submissions has 
not been seen (Harvey, Coulson, Mackaway, & Winchester-Seeto, 2010). There is debate over 
whether reflective practice should be graded (Boud, 1999). There is also a lack of evidence- 
based findings relating to the actual effectiveness of using WIL-specific reflective practice 
assessment protocols to grade student achievement (Boud & Knights, 1996; Jonsson & 
Svingby, 2007; Harvey et al., 2010; Koole et al., 2011). We argue that despite the current lack 
of consensus, and in part because of the momentum weight assessment holds in higher 
education, judgments around the demonstration of reflective practice, in a reflective journal 
for instance, need not be avoided by WIL educators (Moon, 2006). 

Koole et al., (2011) respond to the current lack of consensus on how to assess reflection and 
what classifies as adequate reflective practice by suggesting the application of an eclectic 
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model with practical guidelines for including reflective practice in medical curricula. There 
are many current good practice rubrics, models and frameworks that could, with adaptation, 
function as a set of standards for assessing reflections from WIL placement-life experiences 
(Duke & Appleton, 2000; Gezie, Khaja, Chang, Adamek, & Johnsen, 2012; Kember et al., 2008; 
Langer, 2002; Stupans, March, & Owen, 2013; and Wong, Kember, Wong, & Loke, 2001). The 
model from Ash and Clayton (2004) is particularly noteworthy, as it aims to "help instructors 
from diverse disciplines guide students through reflection and meaningful strategies to 
evaluate the learning outcomes expressed in written products of reflection" (p. 138). While 
their three part model simplifies key reflective learning theories, we suggest an all-inclusive 
tool which teases out and expands upon (rather than abridging) elements of reflective 
practice would be of greater benefit and be more effective to educators new to summative 
graded judgments of student achievement of reflective ability. That being said, there are 
indeed challenges associated with constructing marking schemas that encapsulate: the 
individual's process of reflective practice; the depth of detailed authentic explorations and 
critiques of personal experiences; and the learner's attempt to pry open assumptions and 
reach new understandings. It follows that the criteria and aligned levels of reflective practice 
performance must be robust enough to define and demonstrate the particular type of 
reflective practice process required, whilst flexible enough to allow for learner-centric 
personalized reflections applicable to a broad spectrum of placement experiences. 

Thus, the significance of this pilot study is that it tests a schema for measuring the 
demonstration of reflective practice processes within a WIL placement-life unit. Our aim was 
to ascertain if the tool has potential to function as a set of standards likely to assist a wide 
range of WIL educators when seeking to judge student achievement of standards relating to 
reflective practice during placement-life experiences. We base our evidence of effectiveness 
on two key elements: inter-assessor reliability in producing final grades within a range of 
10%; and the meaningfulness and value of the criteria and grade band descriptors in the 
rubric for diagnosing and judging student achievement with a standards-referenced 
assessment instrument. As such, this study is a response to the call from Higgs (2014) for the 
review and refinement of assessment protocols in WIL, in conjunction with the related 
recommendation by Smith, Ferns, Russell, and Cretchley (2014) for a consistent approach in 
current WIL curricula and curriculum standards. Progress is needed towards addressing the 
demand for credentialing student demonstrations of reflective practice performance against 
intended placement learning outcomes in WIL. 

LITERATURE REVIEW 

The literature indicates an increase in interest and rigor in 'reflection' over the last couple of 
decades (Atkins & Murphy, 1993; Heerde & Murphy, 2009). There is a long and extensive 
tradition of reflective learning (Dewey, 1933; Kolb, 1984; Boud et al., 1984; Boud & Walker, 
1990; Brookfield, 1995; Moon, 1999; 2006; Ghaye & Lillyman 2000; Mamede & Schmidt, 2004; 
White, Fook, and Gardner, (2006), but of most significance to WIL is Schon (1983). In his 
pioneering conceptual framework, reflection on professional practice is where meaning is 
made, and reflection linked to authentic learning experiences in turn contributes to improved 
ongoing practice. Today reflection in WIL is proposed to help students make sense of links 
between theory and practice (McGuire, Lay, & Peters, 2009; Higgins, 2011; Peach & Gamble, 
2011; Smith, 2011); enable the re-shaping of tacit knowledge (Connor, 2004; Cowling, 2004); 
initiate lifelong learning (Mezirow, 2006); build capacity in career development learning 
(Leong, 2012); and develop professional agency by learning to adapt to the nuances of WIL 


Asia-Pacific Journal of Cooperative Education, 2016,17(2), 135-147 


136 



YOUNG, JAMES, NOY: Exploration of a reflective practice rubric 


placement experiences (Billet, 2009). Reflective practice is seen as that which enables personal 
experiences of knowledge transformation (Kalantzis & Cope, 2014; Leonardo, 2004). 

Non-traditional modes of assessment such as the demonstration of reflective practice 
through journals and portfolios, tend to be the preferred option for evidencing personalized 
learning (Ghaye, 2007, p. 151). There are canonized debates around how to, and indeed, if 
reflection should be summatively assessed and graded usually due to this personal nature of 
the explorations and responses to experiences demonstrated in journals (Sumsion & Fleet, 
1996). Indeed, marking reflection can be fraught with issues, in particular because "the 
assessment discourse celebrates certainty; reflection thrives on doubt" (Boud, 1999, p.123). 
Questions relating to the appropriateness of grading portfolios and journals which raise 
ethical tensions around privacy and exposing the self means that the student author might be 
torn between "writing personal and safe responses"; their "views and the views of significant 
others"; as well as decisions to write about "the particular and the general" (summarized by 
Ghaye, 2007, p. 157-158). We acknowledge that reflection artefacts are more difficult to 
assess than yes/no knowledge constructs. This is because reflective practice "is about 
learners constructing their own meanings within a community of professional discourse" 
(Boud, 1999, p. 123). 

However we argue that it is not only feasible but desirable to judge (and therefore 
summatively grade) the process of discovery and of learning within artefacts. The 
assessment driven culture of higher education (HE) contributes to the complexity around the 
assessment of reflective practice. Boud (1999) claims that it is inappropriate to place a value 
on student reflections. He proposes that when it comes to reflection tasks and assessments, 
educators should re-think the all-pervasive broader context and problematize the status quo 
of the current assessment driven culture of HE (Boud 1999, p. 127) and encourages educators 
developing good reflective practice to question whether the reflective activities should be 
part of the formal curriculum (1999, p. 124). Conversely, the framing of thinking in this 
study is to remain within the current assessment driven environment and posit a practical 
way in which educators can function within the status quo of the grade-oriented landscape. 
In general, summative assessment has strategic benefits for the educator operating in an 
assessment-driven curriculum because students put effort into that which is to be assessed 
(and hopefully learn along the path of demonstrating what they have learned (Ramsden, 
2003). The inclusion of a grading system for the assessment of a reflection item provides 
tracking feedback to students as well as motivation to perform, as supported by research that 
shows that student motivation is seen to be heightened when grades are attributed to 
outstanding performance (Kneale, 1997; Sadler, 2009). 

A further challenge relates to upholding the valuable tenets of constructive alignment as 
good practice when designing curricula (Biggs, 1996). Following curriculum development 
principles of constructive alignment, the choice of assessment activities likely to maximize 
student learning is a critical consideration, as are consistent and reliable approaches for 
marking that accommodate the uniqueness of each placement, the individual student 
experience and the raft of possible learning outcomes inherent in placement-//^ experiences. 
Rubrics are useful for documenting the standards for measuring student learning, both for 
defining the expectations of the assessment learning outcomes as well as for marking the 
artefact submitted (Boud & Dochy, 2010). Before this can be achieved within the WIL HE 
community, the standards for reflective practice in WIL require in-principle agreement. 
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Smith et al. (2014) suggest that in WIL, the "integration of theory and practice aligned to 
learning outcomes and assessment should form the basis of curriculum design" (p. 8). 

Issues of standards are addressed in two large studies (Kember, Leung, et al„ 2000; Kember, 
McKay, Sinclair, & Wong, 2008) who tested a reflection questionnaire to ascertain the level of 
reflective thinking by nursing students. Lethbridge, Andrusyszyn, Iwasiw, Laschinger, and 
Fernando (2013) further tested the instrument, claiming that the questionnaire was a "helpful 
instrument to researchers and educators to assess the extent to which students engage in 
reflective thinking during their educational programme" (p. 321). They concluded that while 
the empirical evidence of learning was not part of the study, "Kember and Leung's (2000) 
questionnaire appears to be a reliable and valid instrument to measure reflective thinking" 
(p. 323). The four level schema trialed ranks students' achievement of reflective practice 
constructs, with the fourth level (critical reflection) appearing to indicate a grade descriptor 
of exceeding expectations. This supports the thinking in this paper relating to the need for 
prescribed standards for grading work. However, a review of wider studies on the reliability 
and effectiveness of rubrics in general, beyond 'WIL and reflective practice' found that while 
the use of rubrics in higher education is widespread, there are surprisingly few empirical 
studies that report on rubrics being used for performance assessment (Jonsson & Svingby, 
2007). Jonsson and Svingby (2007) conclude that rubrics can enhance the reliability of 
judging and scoring student performance especially when the rubrics are analytic, specific to 
the task, had exemplars attached and are rolled out with assessor training (p. 141). They add 
that despite this "rubrics do not facilitate valid judgment of performance assessments per se" 
(p. 141). 

In this way, we focus our investigations on whether a "shared understanding" of an 
assessment tool (via assessor-team discussions prior to, during and post marking) is still 
needed in spite of seemingly robust frameworks and understandings of them. The pilot 
therefore considered if a new marking instrument could be used instinctively and with 
effectiveness to the assessors in terms of producing consistent intra and inter-assessor results. 
This study tests the instrument's reliability for positioning student work to grade bands; it 
also investigates two further areas: first, a schema that fits into current 10% grade bands, and 
second, tests whether reliability can be an indicator of transferability of the tool for use across 
WIL contexts. 

THE RESEARCH 

The research explores the validity, reliability and transferability of a reflective practice 
teaching and assessment resource titled 'PRAC'. PRAC is an acronym for a reflective 
practice model: 'P' is for the 'process' of exploration the learner undertook when 
approaching and refining their reflective practice; 'R' is for the 'reporting' of experiences 
selected for reflective purposes; 'A' is for the 'analysis' of those experiences; and finally, 'C' is 
for the plan for 'continual learning'. The pedagogical foundation of PRAC - as is the case for 
the learning framework used by Ash and Clayton (2004) - is the 'What, So What and Now 
What?' reflection model (Borton, 1970). 

PRAC is an analytic rubric based on criterion reference testing for assessing WIL placement- 
like experiences. The rubric consists of seven differently weighted criteria for assessment 
(vertical axis) and six aligned levels of achievement (horizontal axis). PRAC arose when 
current frameworks were deemed as not fit-for-purpose when assessing reflective practice in 
WIL placements (Duke & Appleton, 2000; Gezie et al., 2012; Kember et al., 2008; Langer, 2002; 
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Stupans et al., 2013; and Wong et al., 2001; Ash & Clayton, 2004). PRAC is a re¬ 
conceptualized approach for assessing reflective practice based on the combination of models 
that focus both on the process and levels of reflection. As such, PRAC is not a new model per 
se, but rather, a synthesis of standards and hierarchical levels of reflective process thinking. 
The criteria in PRAC have been validated through wide acceptance and use across discipline 
areas in the reflective practice curricula: the iterative processes of reflection as derived from 
Schon (1983) and Boud et al. (1984). The iterative reflective practice process appears in the 
vertical axis of PRAC as: 

• Seven criteria: process of reflective practice; report on the experience; habits and 
behaviours; emotion; point of view; prior experiences; and continual learning. 

• Adaption of Borton's (1970) What, So What, Now What? model groups the seven 
criteria to assist in the phasing of the overarching process. 

• Inclusion Boud et al.'s (1984) four levels of reflection within Borton (1970): the 
What? phase involves the descriptor 'return to the experience'; the So What? phase 
uses the descriptors 'attending to feelings' and 're-evaluating the experience'; the 
Now What? uses outcomes/resolutions' as the descriptor for that aspect of the 
iterative phase. 

• Questions and prompts accompany each criterion (i.e., Process: Do you refer to time 
and type of process/model used to reflect during and post placement?) 

• Weighting of each criterion (i.e., Process: 10%; Point of View (20%) 

The levels in PRAC, adapted mostly from Moon (1999; 2006), also serve as a validation 
process due to the acceptance and use of her work across disciplines in teaching and 
assessing reflection. 


TABLE 1: Achievement levels in PRAC with accompanying descriptors that expand on the 
performance of achievement for criteria 


Fail: 

Pass: 

Credit: 

Distinction: 

High Distinction: 

0-49 

50-59 

60-69 

70-79 

80-94 

95+ 

Not yet to a 

Returns & 

Connects to 

Synthesizes to 

Revises and 

Looping 

pass standard 

observes to 

make sense 

make 

re-evaluates 

deconstruction 

of reporting 

tell 


’meaning 1 

for new 

/reconstruction 

on 




meanings 

to transform 

observations 







METHODOLOGY 

A sequential mixed methods design was selected to explore the reliability, validity and 
transferability of the generic WIL rubric PRAC. First by capturing quantitative data based on 
inter-assessor agreement/disagreement of the quality of student journal submissions, 
followed by qualitative data collection relating to assessor perception of the use, value and 
meaningfulness of the rubric to function as an assessment tool across a range of WIL 
placement-///re offerings. The research aimed to resolve whether the rubric used in this case 
study has potential as an assessment innovation in WIL reflective practice curricula. 

The PRAC rubric was created by the lead researcher of this study for use in a separate 
placement unit prior to this pilot project - this researcher was not involved in the teaching or 
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marking of the study tour unit. The other two researchers in this study teach into the study 
tour unit and had used an alternate rubric to mark the journals prior to the pilot. For the 
purposes of this study, all 3 researchers used PRAC to test if it had potential as a generic 
template (set of analytic standards) for assessing reflective practices in a range of WIL 
placement-hle curricula. One researcher was new to the journals and familiar with the 
rubric, and two researchers were new to the rubric but had graded the journals using a 
different assessment rubric. They only graded the student journals of those students who 
volunteered to be part of the research. 

The teaching unit used was an international study tour unit. The unit was deemed an 
appropriate case study for data capture because it was not formally classified as a placement 
unit, and as such was significant to the research aim of discerning if PRAC had the potential 
to function reliably across WIL placement -/ike contexts. PRAC was not reviewed or adjusted 
in any way to ensure constructive alignment of the rubric with the case study's assessment 
and unit learning outcomes. This decision was made in order to better investigate base level 
of transferability of PRAC to WIL placement-Lfe experiences. 

Ethics approval for the project included requesting consent for access to previously marked 
reflective journals from 40 second/third year students (2011-2013 cohorts). Eleven students 
(w=ll) agreed for their journals to be marked by three assessors (n=3). The assessors were the 
two academics that lead one or more of the study tours, and the educational researcher 
responsible for the design, development and implementation of PRAC. The assessors, also 
the researchers in this paper, will be referred to as assessors 1, 2 and 3. An initial briefing 
about the components of the rubric was followed by a pre-test marking of one journal to 
calibrate marking levels. All 11 de-identified journals were marked using PRAC. Qualitative 
data capture involved a thematic analysis of responses by the assessors to open-ended 
questions, completed after marking the 11 journals. These questions explored the assessors' 
perceptions of whether PRAC had appropriate criteria and levels for judging the 
performance of reflective practice, and whether the rubric indicated potential for use across 
multiple WIL contexts. Group discussions post data collection provided further opportunity 
to interrogate findings, divergent views, and to reflect on the rubric's transferability. 

FINDINGS 

Quantitative Data Analysis 

There was a wide range in the marks awarded by the assessors, with the 'overall mark' 
showing a range of greater than 11% for the three assessors for each of the eleven journals 
(Table 1). A range of up to 10% between assessors was agreed to provide a demonstration of 
reliability across a small assessment team; this is the range most tertiary institutions use to 
represent assessment grades. As all of the journals had a mark range of greater than 11%, 
this demonstration of reliability had not been achieved. 

Further analysis shows that three of the seven criteria were responsible for most of the 
divergence between assessors. For four of the seven criteria (experience, emotion, point of 
view and prior experience) there was reasonable inter-assessor agreement with a range of < 
20% (Table 2). A much greater range was seen for the other three criteria (process, habits and 
continual learning). When the three criteria in which inter-assessor agreement was poorest 
(process, habits and continued learning) were removed from the calculation, the range for the 
agreement between the three assessors was less than 10% for six of the journals (Table 2). 
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TABLE 2: Number of journals and criteria within different mark ranges 


Criteria 

Range in 
assessor's marks 

of between 0-10% 

Range in 
assessor's marks 
of between 11- 

20% 

Range in 
assessor's marks 

of >20% 

Overall mark (incorporating 
all 7 criteria) 

0 

4 

7 

Experience 

3 

6 

2 

Emotion 

0 

9 

2 

Point of view 

1 

8 

2 

Prior experience 

3 

6 

2 

Process 

0 

0 

11 

Habits 

2 

0 

9 

Continual learning 

0 

2 

9 

Mark (excluding the criteria 
of process, habits, and 
continual learning) 

6 

3 

2 


The range for three of the journals was between 10- 20%, and the range for the remaining two 
journals was less than 30%. This improved level of inter-assessor agreement indicates that 
the majority of PRAC's criteria are reliable in enabling consistent marking across an 
assessment team. The qualitative investigation explored why the remaining criteria did not 
result in inter-assessor agreement. 

Qualitative Data Analysis 

While the quantitative data analysis indicated areas of limited reliability and validity within 
the instrument and limited potential claims about PRAC's transferability as a WIL 
placement -/ike reflective practice assessment tool, the qualitative data tells a different story. 

In their individual written responses, completed after assessing the journals, all assessors 
found the generic rubric useful for providing logical grade levels and practical criteria for 
demonstrating the iterative process of reflection. Specific suggestions relating to the wording 
and structure of the instrument were generally minor, thus inferring general assessor 
agreement on the validity of the rubric. All three felt that PRAC would most likely be of 
value to assessing reflections in a wide range of placement -/ike experiences. This comment, 
in spite of the empirical data (which at the time of qualitative feedback was unknown to the 
assessors) indicates reasonable assessor agreement on the transferability of the instrument. 

The assessors new to the rubric reported that it took practice to become familiar with the 
detail of the rubric, and that it would be difficult for less experienced assessors to work with 
its complexity. Nevertheless, they commented that once the matrix was understood, the 
converging elements in the X and Y axes did not overcomplicate the marking but provided a 
structure that signposted elements to look for in the journal, and the levels/standards for 
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'scoring'. The 'complicated' rubric was stated to be 'a complex but manageable and useful 
marking tool' for assessors familiar with reflective practice concepts. 

When presented with the post-marking empirical data, assessors were surprised at the 
degree of discrepancy in the results. While the pre-test marking of one journal had seemed 
to provide a shared understanding of the seven criteria and the corresponding levels, the 
post-marking discussions prompted the assessors to delve further into what each criterion 
and associated level meant to each assessor. This discussion identified subtle discrepancies 
in shared understandings and thus was identified as a major factor surrounding the lack of 
consistency in the quantitative results. The three assessors undertook two different marking 
process tactics to ascertain if PRAC had merit as a transferable tool. One tactic was to 
deliberately construe the criteria wording broadly and loosely according to pre-existing 
marking mindsets and frameworks based on interpretation. An alternate tactic was to 
deliberately apply each criterion wording definitively and adhering to a research mindset 
and thus avoiding interpretation. 

DISCUSSION 

Shared understanding of the process and aim of assessment is critical from the onset (Boud & 
Dochy, 2010); the incongruence of approach around if and when to interpret the rubric had a 
significant impact on the lack of consistency in grading the 11 journals. However despite the 
disparity in marking that resulted from this difference in assessor application of the rubric, 
the research suggests that assessors found value in both the criteria and levels elements of 
the rubric. This supports Moon's approach which is to provide reflective practice educators 
with iterative elements and levels of reflective practice to determine if students have gone 
beyond just meeting or not meeting standards (Moon, 2006). There are studies that dispute 
this and argue for simplified rating scales rather than grade descriptors for ease of use 
(O'Donovan, Price, & Rust, 2004). While the quantitative results of this study might suggest 
that pass/fail grades are more reliable and easier to judge, the qualitative data indicates that 
the detail provided in the grade bands was valuable for judging the level of performance. 

Assessors felt that a critical reason for the lack of consistency in results was the lack of 
alignment between PRAC and unit learning outcomes. The three most unreliable criteria, 
(process, habits, and continual learning) were poorly aligned with the assessment task and 
accounted for the largest proportion of the discrepancy between the assessors. One assessor 
noted that a generic rubric designed for traditional WIL placements was 'too much of a 
stretch' when applied to intensive experiences, and that criteria should align with subject 
goals and student instructions. This is supported by Moon (2006) and Smith (2011) who 
point to the need for clarity of purpose and alignment of criteria with assessment objectives. 

Given that this study deliberately omitted good practice marking protocols (such as 
curriculum alignment and enacted processes of non-linear marking) from the research design 
to ascertain if the instrument functioned appropriately 'as-is', it is reasonable that not all data 
fully supported claims of reliability. However, the data indicate the value of PRAC as an 
innovation of process rather than as a final product for implementation. In other words, it 
has merit in terms of transferability if it is understood as requiring adaptation to context. 
This exploratory study supports the claim by Jonsson and Svingby (2007) who concluded 
from their study that scoring rubrics can improve holistic assessment frameworks and 
enhance reliable assessment around complex judgments of performance if they are analytical 
and specific to the task. 
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Two further explanations are offered for the discrepancy of results. The first is the adherence 
to the marking protocol of the study, which omitted both the individual marker's ability to 
re-read, re-view and possibly re-mark the first few journals assessed. Brown, Bull, and 
Pendlebury (1997) state that the "major threat to reliability is the lack of consistency of an 
individual marker" (p. 235). The second is the diversion from the normal practice of cross¬ 
marking to benchmark inter-assessor standards. Rubrics perform best and as they are 
intended when "we compare them to published standards, show them to another teacher, or 
ask a colleague to co-score some student work" (Andrade, 2000 p. 30). However 
collaboration may not eradicate skewed interpretation, judgment or subjectivity, as these 
phenomena are inherent in all marking. With rubrics, intra-assessor and inter-assessor bias 
can only be reduced by improving both the quality of the instrument and the team process of 
using the rubric (Crisp, 2008). The non-linear process was not part of the study's marking 
protocol to ascertain whether the marking review step could be bypassed when using PRAC. 
Both the qualitative and quantitative data do not indicate that PRAC was able to justify the 
omission of that step in the marking process. 

Although the assessors' different approaches (interpretative and definitive) to analyzing a 
rubric for research purposes lead to skewed results, the variations enabled identification and 
consolidation of the fundamental strengths and weaknesses of the instrument, the process of 
marking with rubrics, and the grading of reflective practice in WIL curricula. As an 
assessment innovation in WIL curriculum, this study finds that the instrument is not 
sufficiently robust to be used across contexts without amendment or the need for some 
degree of assessor interpretation. It is suggested that the tested generic rubric PRAC should 
be reviewed and adapted where appropriate to align with the contextualized task, unit 
specific language and learning outcomes. This should improve the classification of grade 
banded demonstrations of reflective practice artifacts from students. However equally the 
generic template challenges educators to ensure that their learning outcomes contain the key 
elements of reflective practice. 

This study also proposes that supporting resources aimed to improve shared understanding 
are critical for enhancing the validity of future versions of PRAC. Rubrics are always prone 
to interpretation resulting in possible deviations of shared understandings by members of 
assessment teams (O'Donovan et al., 2004, p. 327). Discrepancies are most likely to occur 
where a teaching team lacks shared understanding of the purpose of using rubrics, the 
process of using the actual rubric at hand, and the meaning of each criterion - best elucidated 
by working through actual examples. This proposition is supported by Jonsson and Svingby 
(2007) who concluded that scoring rubrics can improve holistic assessment frameworks if 
accompanied by assessor training (p. 141). 

CONCLUSION 

The pilot study revealed a tension in findings: the quantitative analysis found a lack of 
reliability for some criteria; while the qualitative data analysis highlighted assessor 
agreement about the value of PRAC as an all-inclusive tool. Demonstration of this 
confidence comes from the assessors using a refined version of PRAC in the following study 
tour (note: this occurred post this research project). PRAC provides a practical starting point 
for developing reflective practice assessment innovations needed to measure learning in the 
complex learning spaces typical in WIL placement-Lfe activities. No longer do we need 
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teaching teams in the sector to develop from scratch judgment markers for credentialing 
reflective learning derived from WIL placement-life experiences. 

IMPLICATIONS 

This study responds to the call for depth and breadth in research and practice around 
frameworks that support constructive alignment of WIL activities and assessments to enable 
integrative learning (Smith, 2012, p. 259). Targeted studies like this contribute to enhancing 
consistent WIL placement assessment approaches. Although conceptualizing, designing, 
implementing and testing for reliable and valid measures of and for reflective practice is 
complex, WIL practitioners using reflective practice for showcasing the narrative of learning 
need evidence-based instruments to verify the level of learning gained from students' self¬ 
reflections. WIL curricula, now part of the widening assessment frame in HE, need 
appropriately chosen assessments to help students approach praxis and be rewarded in a 
consistent fashion to assessment protocols in non-WIL units. Within an educational culture 
demanding assurance of standards, the paucity of research in this area needs to be 
addressed. 

This study highlighted strengths and weaknesses in the application of a generic rubric to a 
WIL placement-life experience. The study supports the use of rubrics to assess and grade 
reflective practice artefacts. The study also offers an all-inclusive framework highlighting the 
areas that produced the most reliable and valid inter-assessor agreement. At the very least, 
the criteria in which a 10% marking range occurred suggest that the majority of the 
instrument is worthy of uptake and application. For improved reliability, PRAC should not 
be used as the end point of the assessment process, but as the beginning of a collaborative, 
constructively aligned assessment process that explores: how the generic rubric will address 
curriculum alignment protocols, what constitutes reflective practice (criteria and levels) in 
the unit, weightings around key elements of the type of reflective practice required, and also 
discussion on how the teaching of the rubric for student capacity building will occur. 

As such this study contributes to exploration of the relatively uncharted domain of 
how and what should be assessed in the rich landscape that is reflective practice in 
WIL placements. Overall, this study finds that traditional grading protocols in the 
current assessment driven culture in HE should and can equally be applied to 
reflective learning by using tools like PRAC for enabling the summative grading of 
student reflective practices in WIL placement-life curricula. 
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